<!DOCTYPE html>
<html lang="en">

<head>
  <meta charset="utf-8" />
    
  <meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1" />
  <title>
    Excel导入优化 |  jojoecfy的个人博客
  </title>
  <meta name="generator" content="hexo-theme-yilia-plus">
  
  <link rel="shortcut icon" href="/images/cfy.png" />
  
  
<link rel="stylesheet" href="/css/main.css">

  
<script src="/js/pace.min.js"></script>


  

  

</head>

</html>

<body>
  <div id="app">
    <main class="content">
      <section class="outer">
  <article id="post-excel" class="article article-type-post" itemscope
  itemprop="blogPost" data-scroll-reveal>

  <div class="article-inner">
    
    <header class="article-header">
       
<h1 class="article-title sea-center" style="border-left:0" itemprop="name">
  Excel导入优化
</h1>
  

    </header>
    

    
    <div class="article-meta">
      <a href="/2020/02/25/excel/" class="article-date">
  <time datetime="2020-02-25T11:13:10.000Z" itemprop="datePublished">2020-02-25</time>
</a>
      
      
      
<div class="word_count">
    <span class="post-time">
        <span class="post-meta-item-icon">
            <i class="ri-quill-pen-line"></i>
            <span class="post-meta-item-text"> 字数统计:</span>
            <span class="post-count">1.3k字</span>
        </span>
    </span>

    <span class="post-time">
        &nbsp; | &nbsp;
        <span class="post-meta-item-icon">
            <i class="ri-book-open-line"></i>
            <span class="post-meta-item-text"> 阅读时长≈</span>
            <span class="post-count">4分钟</span>
        </span>
    </span>
</div>

      
    </div>
    

    
    
    <div class="tocbot"></div>





    

    
    <div class="article-entry" itemprop="articleBody">
      


      

      
      <h1 id="导入优化"><a href="#导入优化" class="headerlink" title="导入优化"></a>导入优化</h1><p>工作中遇到一个问题，在进行导入操作时，将大量数据通过excel导入至数据库中时，性能奇差，导入200条数据已是极限，已经远远达不到预期效果，经笔者解决后效率得到大幅提升，由200-300条数据死机到10s内导入2000条以上，基本达到业务要求，今日便与各位分享一下各中心得与问题。</p>
<a id="more"></a>
<p>  基本业务逻辑：</p>
<pre><code>此功能为导入学生信息，导入的学生信息可以为新增数据，也可以为已有数据（根据身份证判断），导入的学生信息包括学生个人信息，学生户籍地、实际居住地信息，学生专业信息，导入时同时还有对未注册学生进行注册账户操作，在导入时记录log</code></pre><p>原有操作逻辑：</p>
<p>从excel读取每一条要导入的dataList ，遍历dataList,在每一条数据中进行如下操作</p>
<p>一、 拿到id判断是插入还是更新，如为更新操作，则省略部分字段校验的字段与更新，插入操作则对每个字段按规则进行校验并创建学生账号。</p>
<p>二、根据数据去数据库中查询户籍地信息是否正确，此处学生的户籍地信息分为省，市，区，具体地址四个字段，主要依据行政区划表查询省市区是否对应，实际居住地逻辑于此相同。</p>
<p>三、学生专业信息包含文化层次、报名方向，报名层次，专业四个字段，而学生信息中包含第一志愿与第二志愿，志愿信息由报名方向，报名层次，专业组成，每一条志愿都须去招生计划表中查询招生计划是否存在，</p>
<p>四、查询推荐教师是否存在（查询user表）</p>
<p>五、向log表中记录log</p>
<p>六、保存（更新）记录</p>
<p>这个项目为老项目，从前开发者为了省力直接调用已有api，哪怕十分冗余，大致看去，很多数据库相关操作可以放到循环外执行，只是为了方便把数据库操作放到循环里，所有笔者一开始尽量把有关数据库操作放到了循环外。但即便这样实现效果依旧不理想。经过一段时间的考虑笔者将代码优化至如下流程。</p>
<p>1、创建datalist，优化初始空间，由于理论上单次导入学生不会超过10000条，直接将list初始化容量控制在这个范围，</p>
<p>从数据库中查出学生集合，按此容量创建并填充idmap</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">    Map&lt;String, ZsStudent&gt; identityCardMap </span><br><span class="line">&#x3D; new HashMap&lt;String, ZsStudent&gt;((int) (dbStuList.size() &#x2F; 0.75) + 1);</span><br></pre></td></tr></table></figure>
<p>避免多次rehash降低效率，id为key, student 为value，将excel数据放入list———datalist</p>
<p>2、查询所有有关招生计划，转为list，值为文化层次、报名方向，报名层次，专业拼接——招生list</p>
<p>  在用户登录时查询省市区值，将其放入redis—-地区缓存（分为3个map）（先删后存）</p>
<p>推荐教师读取其他功能配置，放入缓存—–教师缓存</p>
<p>2、创建insertList，updateList，每个list3个，每个list的size控制在3000，遍历dataMap，如为插入数据则将其放入insertList，update同理</p>
<p>遍历map时按与招生计划一样的规则填充list—–志愿list，遍历datalist时通过地区缓存判断学生地区是否正确，通过教师缓存判断教师是否为所配置推荐教师，如异常则break跳出循环，返回提示,datalist遍历完毕。</p>
<p>3、判断招生list是否存在志愿list？“”：返回提示；</p>
<p>4、log不在单独记录，只记录共读取数据数。</p>
<p>5、创建中间表，此表有所有所需要的字段，将updateList放入此表，将此表更新至各表，删除此表，</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br></pre></td><td class="code"><pre><span class="line">  &#x2F;*创建临时表*&#x2F;</span><br><span class="line">   void creatStudentAndUserTemp();</span><br><span class="line"></span><br><span class="line">   &#x2F;*需要更新的学生数据插入临时表*&#x2F;</span><br><span class="line">   void insertStudentAndUser(List&lt;ZsStudent&gt; updateStudentLists);</span><br><span class="line"></span><br><span class="line">   &#x2F;*根据临时表更新xg_student表*&#x2F;</span><br><span class="line">   void updateStudent();</span><br><span class="line"></span><br><span class="line">   &#x2F;*根据临时表更新sys_user表*&#x2F;</span><br><span class="line">   void updateUser();</span><br><span class="line"></span><br><span class="line">   &#x2F;*删除临时表*&#x2F;</span><br><span class="line">   void dropStudentAndUserTemp();</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">   void insertStudent(List&lt;ZsStudent&gt; insertStudentLists);</span><br><span class="line"></span><br><span class="line">   &#x2F;*根据临时表更新sys_user表*&#x2F;</span><br><span class="line">   void insertUser(List&lt;ZsStudent&gt; insertStudentLists);</span><br><span class="line"></span><br><span class="line">   &#x2F;*学生导入时-批量插入用户角色关联数据*&#x2F;</span><br><span class="line">   void insertImpportUserRole(@Param(&quot;list&quot;) List&lt;ZsStudent&gt; </span><br><span class="line">insertStudentLists,@Param(&quot;roleId&quot;) String roleId);</span><br><span class="line"></span><br><span class="line">   &#x2F;*根据临时表更新volunteer表*&#x2F;</span><br><span class="line">   void updateVolunteer();</span><br><span class="line">   &#x2F;*插入volunteer表*&#x2F;</span><br><span class="line">   void insertVolunteer(List&lt;ZsStudent&gt; insertStudentLists);</span><br></pre></td></tr></table></figure>
<p>insertList走branchInsert方法插入数据库。</p>
<p>上述即为优化后的基本逻辑，主要将需多次查询数据库操作放入缓存，尽可能减少数据库操作，减少循环中业务操作，</p>
<p>目前方案减少了数据库操作是因为将数据库操作分散，放入缓存，在导入大数据量数据时会占用大量内存（hashmap），但这是目前比较可以接受的方案，经过优化导入功能整体的性能也得到了提高。</p>

      
      <!-- reward -->
      
    </div>
    
    
      <!-- copyright -->
      
        <div class="declare">
          <ul class="post-copyright">
            <li>
              <i class="ri-copyright-line"></i>
              <strong>版权声明： </strong s>
              本博客所有文章除特别声明外，均采用 <a href="https://www.apache.org/licenses/LICENSE-2.0.html" rel="external nofollow"
                target="_blank">Apache License 2.0</a> 许可协议。转载请注明出处！
            </li>
          </ul>
        </div>
        
    <footer class="article-footer">
      
      
  <ul class="article-tag-list" itemprop="keywords"><li class="article-tag-list-item"><a class="article-tag-list-link" href="/tags/java/" rel="tag">java</a></li><li class="article-tag-list-item"><a class="article-tag-list-link" href="/tags/%E4%B8%9A%E5%8A%A1/" rel="tag">业务</a></li></ul>


    </footer>

  </div>

  
  
  <nav class="article-nav">
    
      <a href="/2020/02/27/linux/" class="article-nav-link">
        <strong class="article-nav-caption">上一篇</strong>
        <div class="article-nav-title">
          
            常用Linux命令
          
        </div>
      </a>
    
    
      <a href="/2020/02/25/suoyin/" class="article-nav-link">
        <strong class="article-nav-caption">下一篇</strong>
        <div class="article-nav-title">数据库索引概述</div>
      </a>
    
  </nav>


  

  
  
<!-- valine评论 -->
<div id="vcomments-box">
    <div id="vcomments">
    </div>
</div>
<script src="//cdn1.lncld.net/static/js/3.0.4/av-min.js"></script>
<script src='https://cdn.jsdelivr.net/npm/valine@1.3.10/dist/Valine.min.js'></script>
<script>
    new Valine({
        el: '#vcomments',
        app_id: 'hVXA4YKowVsvPVBi9VpCSsFP-gzGzoHsz',
        app_key: 'TowHJn0OK2oMVqgOPiONsbaT',
        path: window.location.pathname,
        notify: 'false',
        verify: 'false',
        avatar: 'identicon',
        placeholder: '给我的文章加点评论吧~',
        recordIP: true
    });
    const infoEle = document.querySelector('#vcomments .info');
    if (infoEle && infoEle.childNodes && infoEle.childNodes.length > 0) {
        infoEle.childNodes.forEach(function (item) {
            item.parentNode.removeChild(item);
        });
    }
</script>
<style>
    #vcomments-box {
        padding: 5px 30px;
    }

    @media screen and (max-width: 800px) {
        #vcomments-box {
            padding: 5px 0px;
        }
    }

    #vcomments-box #vcomments {
        background-color: #fff;
    }

    .v .vlist .vcard .vh {
        padding-right: 20px;
    }

    .v .vlist .vcard {
        padding-left: 10px;
    }
</style>

  

  
  
  

</article>
</section>
      <footer class="footer">
  <div class="outer">
    <ul class="list-inline">
      <li>
        &copy;
        2020
        jojoecfy
      </li>
      <li>
        
      </li>
    </ul>
    <ul class="list-inline">
      <li>
        
        
        <span>
  <i>PV:<span id="busuanzi_value_page_pv"></span></i>
  <i>UV:<span id="busuanzi_value_site_uv"></span></i>
</span>
        
      </li>
      
      <li>
        <!-- cnzz统计 -->
        
        <script type="text/javascript" src='https://s9.cnzz.com/z_stat.php?id=1278657708&amp;web_id=1278657708'></script>
        
      </li>
    </ul>
  </div>
</footer>
      <div class="to_top">
        <div class="totop" id="totop">
  <i class="ri-arrow-up-line"></i>
</div>
      </div>
    </main>
    <aside class="sidebar">
      <button class="navbar-toggle"></button>
<nav class="navbar">
  
  <div class="logo">
    <a href="/"><img src="/images/cfy.png" alt="jojoecfy的个人博客"></a>
  </div>
  
  <ul class="nav nav-main">
    
    <li class="nav-item">
      <a class="nav-item-link" href="/">主页</a>
    </li>
    
    <li class="nav-item">
      <a class="nav-item-link" href="/archives">归档</a>
    </li>
    
    <li class="nav-item">
      <a class="nav-item-link" href="/tags">标签</a>
    </li>
    
  </ul>
</nav>
<nav class="navbar navbar-bottom">
  <ul class="nav">
    <li class="nav-item">
      
      <a class="nav-item-link nav-item-search"  title="Search">
        <i class="ri-search-line"></i>
      </a>
      
      
    </li>
  </ul>
</nav>
<div class="search-form-wrap">
  <div class="local-search local-search-plugin">
  <input type="search" id="local-search-input" class="local-search-input" placeholder="Search...">
  <div id="local-search-result" class="local-search-result"></div>
</div>
</div>
    </aside>
    <div id="mask"></div>

<!-- #reward -->
<div id="reward">
  <span class="close"><i class="ri-close-line"></i></span>
  <p class="reward-p"><i class="ri-cup-line"></i>请我喝杯咖啡吧~</p>
  <div class="reward-box">
    
    
  </div>
</div>
    
<script src="/js/jquery-2.0.3.min.js"></script>


<script src="/js/jquery.justifiedGallery.min.js"></script>


<script src="/js/lazyload.min.js"></script>


<script src="/js/busuanzi-2.3.pure.min.js"></script>


<script src="/js/share.js"></script>



<script src="/fancybox/jquery.fancybox.min.js"></script>







<script src="/js/tocbot.min.js"></script>

<script>
  // Tocbot_v4.7.0  http://tscanlin.github.io/tocbot/
  tocbot.init({
    tocSelector: '.tocbot',
    contentSelector: '.article-entry',
    headingSelector: 'h1, h2, h3, h4, h5, h6',
    hasInnerContainers: true,
    scrollSmooth: true,
    scrollContainer:'main',
    positionFixedSelector: '.tocbot',
    positionFixedClass: 'is-position-fixed',
    fixedSidebarOffset: 'auto',
    onClick: (e) => {
      $('.toc-link').removeClass('is-active-link');
      $(`a[href=${e.target.hash}]`).addClass('is-active-link');
      $(e.target.hash).scrollIntoView();
      return false;
    }
  });
</script>


<script>
  var ayerConfig = {
    mathjax: false
  }
</script>


<script src="/js/ayer.js"></script>


<script src="https://cdn.jsdelivr.net/npm/jquery-modal@0.9.2/jquery.modal.min.js"></script>
<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/jquery-modal@0.9.2/jquery.modal.min.css">


<!-- Root element of PhotoSwipe. Must have class pswp. -->
<div class="pswp" tabindex="-1" role="dialog" aria-hidden="true">

    <!-- Background of PhotoSwipe. 
         It's a separate element as animating opacity is faster than rgba(). -->
    <div class="pswp__bg"></div>

    <!-- Slides wrapper with overflow:hidden. -->
    <div class="pswp__scroll-wrap">

        <!-- Container that holds slides. 
            PhotoSwipe keeps only 3 of them in the DOM to save memory.
            Don't modify these 3 pswp__item elements, data is added later on. -->
        <div class="pswp__container">
            <div class="pswp__item"></div>
            <div class="pswp__item"></div>
            <div class="pswp__item"></div>
        </div>

        <!-- Default (PhotoSwipeUI_Default) interface on top of sliding area. Can be changed. -->
        <div class="pswp__ui pswp__ui--hidden">

            <div class="pswp__top-bar">

                <!--  Controls are self-explanatory. Order can be changed. -->

                <div class="pswp__counter"></div>

                <button class="pswp__button pswp__button--close" title="Close (Esc)"></button>

                <button class="pswp__button pswp__button--share" style="display:none" title="Share"></button>

                <button class="pswp__button pswp__button--fs" title="Toggle fullscreen"></button>

                <button class="pswp__button pswp__button--zoom" title="Zoom in/out"></button>

                <!-- Preloader demo http://codepen.io/dimsemenov/pen/yyBWoR -->
                <!-- element will get class pswp__preloader--active when preloader is running -->
                <div class="pswp__preloader">
                    <div class="pswp__preloader__icn">
                        <div class="pswp__preloader__cut">
                            <div class="pswp__preloader__donut"></div>
                        </div>
                    </div>
                </div>
            </div>

            <div class="pswp__share-modal pswp__share-modal--hidden pswp__single-tap">
                <div class="pswp__share-tooltip"></div>
            </div>

            <button class="pswp__button pswp__button--arrow--left" title="Previous (arrow left)">
            </button>

            <button class="pswp__button pswp__button--arrow--right" title="Next (arrow right)">
            </button>

            <div class="pswp__caption">
                <div class="pswp__caption__center"></div>
            </div>

        </div>

    </div>

</div>

<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/photoswipe@4.1.3/dist/photoswipe.min.css">
<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/photoswipe@4.1.3/dist/default-skin/default-skin.css">
<script src="https://cdn.jsdelivr.net/npm/photoswipe@4.1.3/dist/photoswipe.min.js"></script>
<script src="https://cdn.jsdelivr.net/npm/photoswipe@4.1.3/dist/photoswipe-ui-default.min.js"></script>

<script>
    function viewer_init() {
        let pswpElement = document.querySelectorAll('.pswp')[0];
        let $imgArr = document.querySelectorAll(('.article-entry img:not(.reward-img)'))

        $imgArr.forEach(($em, i) => {
            $em.onclick = () => {
                // slider展开状态
                // todo: 这样不好，后面改成状态
                if (document.querySelector('.left-col.show')) return
                let items = []
                $imgArr.forEach(($em2, i2) => {
                    let img = $em2.getAttribute('data-idx', i2)
                    let src = $em2.getAttribute('data-target') || $em2.getAttribute('src')
                    let title = $em2.getAttribute('alt')
                    // 获得原图尺寸
                    const image = new Image()
                    image.src = src
                    items.push({
                        src: src,
                        w: image.width || $em2.width,
                        h: image.height || $em2.height,
                        title: title
                    })
                })
                var gallery = new PhotoSwipe(pswpElement, PhotoSwipeUI_Default, items, {
                    index: parseInt(i)
                });
                gallery.init()
            }
        })
    }
    viewer_init()
</script>




<script type="text/javascript" src="https://js.users.51.la/20544303.js"></script>

    
  </div>
</body>

</html>